Remote gaming applications, in which a server-side game is controlled by a client-side player, have attempted to encode the video output from a three-dimensional (3D) graphics engine in real-time using existing or customized encoders. However, the interactive nature of video games, particularly the player feedback loop between video output and player input, makes game video streaming much more sensitive to latency than traditional video streaming. Existing video coding methods can trade computational power, and little else, for reductions in encoding time. New methods for integrating the encoding process into the video rendering process can provide significant reductions in encoding time while also reducing computational power, improving the quality of the encoded video, and retaining the original bitstream data format to preserve interoperability of existing hardware devices.
Existing video coding standards have only color and temporal information contained in image sequences to improve video encoding time, size, or quality. Some coding standards, such as those in the MPEG standard series, use a computationally intensive block-based motion estimation method to approximate object movement based on the color data contained in a video. These block-based motion estimation methods have historically provided significant reductions in the size of encoded video, but are a source of significant latency in real-time video streaming environments.
Integrating the encoding process into the video rendering process provides access to additional data sources that can be leveraged for encoding improvements. For instance, some 3D graphics engines, such as those contained in a game engine, may already generate motion vectors that perfectly describe the movement of each pixel on each video frame. By providing both the final rendered frame and injecting properly formatted motion vector data into the encoder, the most computationally-complex and time-consuming step in the video encoder, motion estimation, can be skipped for each inter frame. Additionally, the motion vectors supplied by the graphics engine will be more accurate than those approximated by a block-based motion estimation algorithm, which will improve the quality of the encoded video.
These two domains, video encoding and real-time graphics rendering, have traditionally operated separately and independently. By integrating the graphics engine and encoder to leverage the strengths of each, the encoding time can be reduced enough to support streaming applications that are hyper-sensitive to latency.
These and other attendant advantages of the invention will become apparent in view of the deficiencies in the technologies described below.
For example, U.S. Patent Application Publication No. 2015/0228106 A1 (“the '106 Publication”) discloses technology directed to decoding video data to generate a sequence of decoded blocks of a video image. The technology allows for the use of each decoded block of a video image as a separate texture for corresponding polygons of the geometric surface as the decoded block is generated by the codec engine. The '106 Publication technology describes integration between a codec engine that decodes encoded video data to generate the video image to be mapped and a 3D graphics engine that renders the display picture in part by performing the texture mapping of the video image to the geometric surface. However, this technology is deficient compared to the present invention at least because it does not disclose nor use a graphics engine that provides both the final rendered frame and properly formatted motion vector data for injection into the video codec engine, such that the video codec engine does not need to perform any motion estimation prior to transmitting encoded video data to the remote client coding engine. By contrast, the present invention's improvement to computer technology provides reductions in encoding time and computational power, improvement in the quality of the encoded video, and results the retention of the original bitstream data format in order to preserve interoperability.
U.S. Patent Application Publication No. 2011/0261885 A1 (“the '885 Publication”), discloses systems and methods directed to bandwidth reduction through the integration of motion estimation and macroblock encoding. In this system, the motion estimation may be performed using fetched video data to generate motion estimation related information, including motion vectors. These motion vectors may correspond to a current macroblock, using corresponding video data cached in the buffer. Again, the '885 Publication technology is deficient compared to the present invention at least because it does not disclose nor use a graphics engine that provides both the final rendered frame and properly formatted motion vector data for injection into the video codec engine, such that the video codec engine does not need to perform any motion estimation prior to transmitting encoded video data to the remote client coding engine. As such, the technology of the '885 Publication does not provide the same reductions in encoding time and computational power, and the improvement in the quality of the encoded video that the present invention offers.
As is apparent from the above discussion of the state of art in this technology, there is a need in the art for an improvement to the present computer technology related to video encoding in game environments.